Alert rules that let people sleep

A page at 3 AM PT costs about a day of the person's week, counting the hour awake and the day they spend at half speed after it. That makes an alert rule a spending decision, not a safety one.

Four that hold up

• Rate, not existence. More than fifty events in five minutes on a project people care about. One new exception at 4 AM is a morning problem.
• New in this release. An issue first seen after a deploy, on a route with traffic, is the highest-signal page there is — it has a suspect and a rollback attached.
• Regression. A resolved issue that came back. Somebody already decided this one mattered.
• The budget one. A route past its p95 budget for fifteen minutes. Slow is an outage that nobody files a ticket about.

Two that sound responsible

Alert on every new issue. This is the rule teams write in week one and mute in week three, and once a channel is muted it stays muted for the outage too.

Alert on a percentage change. A 300% rise from two events is six events. Every percentage rule needs an absolute floor under it or it will wake somebody up over statistical noise on a Sunday.

The part that is not a rule

Route by ownership, not by rotation. CODEOWNERS and the suspect commit between them know whose change this was better than a schedule does, and a page that arrives at the person who wrote the line last Thursday gets fixed in a fraction of the time.

Then write the escalation down, in the tool, with a timer on it: if nobody acknowledges in ten minutes, it goes to the second person. Teams that keep this in their heads discover on the worst night of the year that two people both assumed the other was looking.